Image generation apparatus, control method therefor, and computer-readable storage medium

ABSTRACT

An image generation apparatus includes: a generation unit that generates an output image in which a foreground mask image overlaps a background image, the foreground mask image having been generated based on a moving object included in an input image; an update unit that executes update processing for updating at least a partial area within the background image based on the input image; and a restricting unit that restricts reflection of the update processing in a restricted area compared to other areas, the restricted area being at least a partial area in the output image.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image generation apparatus, acontrol method therefor, and a computer-readable storage medium, andespecially relates to a technique to generate a privacy protectionimage.

Description of the Related Art

Along with the widespread installation of surveillance cameras,protection of privacy of individuals shown on the video of thesurveillance cameras is becoming increasingly important Japanese PatentLaid-Open No. 2009-225398 describes a technique to protect privacy bygenerating a foreground image and a background image, which respectivelyrepresent an image of people and show no people, from an input image,and making an image, resulting from applying mask processing and filterprocessing to the foreground image, overlap the background image.

Japanese Patent Laid-Open No. 2009-225398 is based on the premise that abackground image is generated in a state where there is no persontargeted for protection. If a background image is generated from thevideo of a surveillance camera in a state where a person exists, thebackground image will include an image of that person. That is to say,the image of that person is distinguished as a background, and is notgenerated as a foreground image. As a result, the privacy of that personmay not be protected. Furthermore, if a person stays motionless for along time from, for example, sitting on a chair, a person area may bereflected in a background image as time passes. In this case also, thatperson area is not treated as an area targeted for protection, and theprivacy of that person cannot be protected.

SUMMARY OF THE INVENTION

One embodiment of the present invention reduces the failure to protectprivacy due to reflection of an image of a person in a background image.

An image generation apparatus according to one aspect of the presentinvention, there is provided an image generation apparatus, comprising:a generation unit that generates an output image in which a foregroundmask image overlaps a background image, the foreground mask image havingbeen generated based on a moving object included in an input image; anupdate unit that executes update processing for updating at least apartial area within the background image based on the input image; and arestricting unit that restricts reflection of the update processing in arestricted area compared to other areas, the restricted area being atleast a partial area in the output image.

According to another aspect of the present invention, there is provideda control method for an image generation apparatus, the control methodcomprising: generating an output image in which a foreground mask imageoverlaps a background image, the foreground mask image having beengenerated based on a moving object included in an input image; executingupdate processing for updating at least a partial area within thebackground image based on the input image; and restricting reflection ofthe update processing in a restricted area compared to other areas, therestricted area being at least a partial area in the output image.

According to another aspect of the present invention, there is provideda computer readable storage medium having stored therein a program forcausing a computer to execute a control method for an image generationapparatus, the control method comprising: generating an output image inwhich a foreground mask image overlaps a background image, theforeground mask image having been generated based on a moving objectincluded in an input image; executing update processing for updating atleast a partial area within the background image based on the inputimage; and restricting reflection of the update processing in arestricted area compared to other areas, the restricted area being atleast a partial area in the output image.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram for describing the principle of generation of aprivacy protection image.

FIG. 2 is a diagram for describing a method for generating a privacyprotection image according to a first embodiment.

FIG. 3 is a block diagram showing an exemplary functional configurationof a video generation apparatus according to the first embodiment.

FIG. 4 is a flowchart showing a procedure of processing executed by thevideo generation apparatus according to the first embodiment.

FIG. 5 is a diagram showing an example of a screen for setting a privacyprotection area.

FIG. 6 is a flowchart showing the operations of setting the privacyprotection area.

FIG. 7 is a diagram showing an example of a screen for setting a privacyprotection area according to a second embodiment.

FIG. 8 is a block diagram showing an exemplary functional configurationof a video generation apparatus according to the second embodiment.

FIG. 9 is a flowchart showing processing for registering an initial maskarea based on human detection.

FIG. 10 is a flowchart showing processing for controlling the speed ofreflection in a background image on a per-mask region basis.

FIG. 11 is a block diagram showing an exemplary hardware configurationof a video generation apparatus.

DESCRIPTION OF THE EMBODIMENTS

The following describes embodiments is detail with reference to theattached drawings. Note that the configurations presented by thefollowing embodiments are merely examples, and the present invention isnot limited to the configurations presented by the embodiments.

<First Embodiment>

According to the present embodiment, in a privacy protection imagegeneration method that silhouettes a detected moving object area andmakes it overlap a background image, an area in which moving objectdetection is difficult is designated in advance. A protection image isprepared in advance by blurring an initial background image in thisarea, and placed to overlap a background image at the time of generationof a privacy protection image. Thereafter, the privacy protection imageis generated by placing a silhouette image through transparent overlap.

FIG. 1 a schematic diagram showing the principle of generation of aprivacy protection image. 11 a, 11 b, 11 c, and 11 d respectively denotean input image 110, a background image 130, a foreground mask image 150,and a privacy protection image 170 that serves as an output image attime t1. Similarly, 12 a, 12 b, 12 c, and 12 d respectively denote aninput image 120, a background image 140, a foreground mask image 160,and a privacy protection image 180 that serves as an output image attime t2.

A person 111 and a person 112 exist in the input image 110 at time t1and the input image 120 at time t2. The person 111 moves between theinput image 110 (time t1) and the input image 120 (time t2). On theother hand, the person 112 is motionless, and his/her position andorientation are kept substantially the same in the input image 110 andthe input image 120.

The background images 130, 140 are generated from a background modelthat has been generated using a known method based on the input images.The person 111 moves in position between time t1 and time t2, and isdistinguished as a moving object. Therefore, an image of the person 111appears in neither of the background images 130, 140. On the other hand,as the person 112 stays motionless from time t1 to time t2, he/she isdistinguished as a background 142 at time t2. That is to say, an objectthat stays motionless (an object that does not move) over apredetermined period is considered a part of a background, and an imageof that object appears in a background image. In other words, the imageof that object is reflected in a background image, and determined as abackground. This predetermined period until reflection in a backgroundimage will be referred to as backgroundization time.

The foreground mask image 150 at time t1 is generated based on the inputimage 110 and the background image 130. For example, the foreground maskimage 150 including foreground masks 151, 152 is obtained by extractingmoving object areas through comparison between the input image 110 andthe background image 130, and converting the extracted moving objectareas into predetermined patterns. In the foreground mask image 160 attime t2, a foreground mask 161 is generated through comparison betweenthe input image 120 and the background image 140. In the foreground maskimage 160, a foreground mask corresponding to the person 112 does notexist because the person 112 is a part of the background image 140 asthe background 142.

The privacy protection image 170 at time t1 is generated throughtransparent overlap of the foreground mask image 150 on the backgroundimage 130. Therefore, in the privacy protection image 170 at time t1,the person 111 and the person 112 are displayed as silhouetted images171 and 172, and their privacy is protected. On the other hand, theprivacy protection image 180 at time t2 is generated through transparentoverlap of the foreground mask image 160 on the background image 140.Therefore, although the person 111 is protected in terms of privacy ashe/she is displayed as a silhouetted image 181, the person 112 is notprotected in terms of privacy and he/she is displayed as the background142.

FIG. 2 is a schematic diagram showing the principle of generation of aprivacy protection image according to the present embodiment for solvingthe foregoing problem (the failure to protect the privacy of the person112). Similarly to FIGS. 1, 21 a, 21 b, 21 c, and 21 d respectivelydenote an input image 210, a background image 220, a foreground maskimage 230, and a privacy protection image 240. A person 211 in the inputimage 210 moves, and does not appear in the background image 220. As aresult, in the foreground mask image 230, a foreground mask 231corresponding to the person 211 is generated. Therefore, in the privacyprotection image 240, the person 211 is displayed as a silhouetted image241, and his/her privacy is protected.

On the other hand, a person 212 within the input image 210 is assumed tobe in a motionless state for a long time (for a period exceeding thebackgroundization time) from, for example, sitting on a chair. Note thatthe motionless state refers to a case where an amount of change in anarea is smaller than a predetermined threshold. In this case, an imageof the person 212 in the input image 210 is reflected in the backgroundimage, and an image 221 of the person 212 appears in the backgroundimage 220. A foreground mask of the person 212 not generated in theforeground mask image 230, and the privacy of the person 212 cannot beprotected.

In the present embodiment, in order to realize protection of the privacyof the person 212, a background mask image 214 is generated by cuttingout a partial area including the person 212 from the input image 210 asa partial image 213. The background mask image 214 is in a state whereprivacy is protected due to application of blurring or mosaic processingthereto. The privacy protection image 240 is generated through overlapof the background mask image 214, and then transparent overlap of theforeground mask image 230, on the background image 220. As a result, theperson 211 is displayed as the silhouetted image 241, and the person 212is included in a mosaicked image 242. Consequently, both of the person211 and the person 212 are protected in terms of privacy.

As described above, in a partial area corresponding to the partial image213 in the privacy protection image 240, reflection of a result of anupdate of the background image (the image 221 that appears in thebackground image 220) is restricted. An area in which reflection of anupdate of the background image is thus reflected will be referred to asa restricted area.

FIG. 3 is a block diagram showing an exemplary functional configurationof a video generation apparatus 300 as an image generation apparatusaccording to the first embodiment. The video generation apparatus 300includes an image obtaining unit 301, a background model update unit302, a foreground background distinction unit 303, a mask imagegeneration unit 304, a background image generation unit 305, an imagecomposition unit 306, an output unit 307, and a parameter setting unit308.

The image obtaining unit 301 obtains a captured image that has beencaptured by an image capturing apparatus as an input image 210. Nolimitation is intended regarding a format and a method in which theinput image is obtained. For example, the input image may be obtained inany format, such as a still image file, a moving image file, and astreaming image, and a method for obtaining the input image may be ofany form using a network, a memory card or other recording mediums, etc.Note that in obtaining a streaming image, a captured image that iscurrently shot by the image capturing apparatus is obtained. In the caseof a still image file and a moving image file, the input image is animage that was shot and stored to a storage apparatus by the imagecapturing apparatus in the past.

The background model update unit 302 executes update processing forupdating a background model based on the input image 210 obtained by theimage obtaining unit 301. The foreground background distinction unit 303distinguishes between a foreground and a background on a per-partialarea basis by comparing an immediately preceding input image and thebackground model. In this distinction processing, image feature amountsof the input image are compared with image feature amounts of thebackground model. Examples of the image feature amounts to be comparedinclude pixel-by-pixel luminance values and block-by-block frequencycomponents, and some methods incorporate multiple types of image featureamounts. This distinction processing is so-called background subtractionprocessing, and background subtraction of any method may be used.

The mask image generation unit 304 generates a foreground mask image 230based on the result of distinction between a foreground and a backgroundby the foreground background distinction unit 303. This is, for example,processing for changing a moving object area (foreground) into aunicolor silhouette. The background image generation unit 305 generatesa background image 220 from the background model. Furthermore, thebackground image generation unit 305 generates a background image inwhich privacy is protected by making a background mask image 214, whichhas been generated from the input image 210 by the parameter settingunit 308, overlap the background image 220 generated from the backgroundmodel. Note that composition of the background mask image 214 may beperformed by the image composition unit 306.

The image composition unit 306 generates a privacy protection imagethrough transparent overlap of the foreground mask image, which has beengenerated by the mask image generation unit 304, on the background imagewith the overlapping background mask image. Alternatively, it blurs aportion corresponding to the foreground mask area in the input image,and places the portion on the background through transparent overlap.The output unit 307 outputs the privacy protection image generated bythe image composition unit 306. Examples of an output method includepresentation on a display, storage to a file, and network distribution.However, an output method used by the output unit 307 is not limited toa particular output method. Note that as mentioned earlier, the imagecomposition unit 306 may perform composition of the background maskimage. In this case, the image composition unit 306 generates theprivacy protection image through transparent overlap of the foregroundmask image on the background image (with no overlapping background maskimage), and through overlap of the background mask image 214 at acorresponding position.

The parameter setting unit 308 sets parameters for the background modelupdate unit 302 and the background image generation unit 305.Furthermore, the parameter setting unit 308 allows a user to set arestricted area by providing a user interface, which will be describedlater using FIG. 5 and the like. The parameter setting unit 308 preparesthe background mask image 214 by obtaining a partial image 213corresponding to the set restricted area from the input image 210 andapplying blur processing to the obtained partial image. As describedearlier, the background image generation unit 305 makes the backgroundmask image 214 overlap the background image 220, or replaces a partialarea corresponding to the restricted area in the background image 220with the background mask image 214.

FIG. 11 is a diagram showing an exemplary hardware configuration of thevideo generation apparatus 300. The video generation apparatus 300includes a CPU 11, a ROM 12, a RAM 13, a display apparatus 14, an inputapparatus 15, an external memory 16, a communication I/F 17, and asystem bus 18.

The CPU 11 integrally controls the operations of the video generationapparatus 300, and controls various components (12 to 17) via the systembus 18. The ROM 12 is a nonvolatile memory that stores control programsand the like that are necessary for the CPU 11 to execute processing.Note that these programs may be stored in the external memory 16 and aremovable storage medium. The RAM 13 functions as, for example, a mainmemory and a working area for the CPU 11. That is to say, the CPU 11realizes, for example, various types of functional units that have beendescribed earlier using FIG. 3 by, at the time of execution ofprocessing, loading necessary programs and the like from the ROM 12 orthe external memory 16 to the RAM 13 and executing these programs andthe like.

The display apparatus 14 is, for example, a quid crystal displayapparatus, and performs various types of display under control of theCPU 11. For example, the display apparatus 14 displays a setting dialogscreen, which will be described later using FIG. 5. The input apparatus15 inputs a user operation to the video generation apparatus 300. Theinput apparatus 15 includes a pointing device, such as a mouse. Theexternal memory 16 stores, for example, various types of data andvarious types of information that are necessary when the CPU 11 executesprocessing using programs. The external memory 16 also stores, forexample, various types of data and various types of information thathave been obtained by the CPU 11 executing processing using programs andthe like.

The communication I/F 17 is an interface for communicating with anexternal apparatus. The communication I/F 17 is, for example, a LANinterface. In the present embodiment, the communication I/F 17 connectsto a network 21, such as a LAN, and obtains captured images from animage capturing apparatus 22 and a server 23 connected to the network21. Note that a network camera, for example, is expected to be used asthe image capturing apparatus 22. The system bus 18 communicablyconnects the CPU 11, ROM 12, RAM 13, display apparatus 14, inputapparatus 15, external memory 16, and communication I/F 17.

The following describes the operations of the video generation apparatus300 configured in the above-described manner.

FIG. 4 is a flowchart showing a procedure for generating a privacyprotection image according to the first embodiment. The procedure forgenerating the privacy protection image will now be described using theflowchart of FIG. 4 and images shown in FIG. 2. First, in step S401, theimage obtaining unit 301 obtains an input image 210. Next, in step S402,the background model update unit 302 updates a background model based onthe obtained input image 210. Subsequently, in step S403, the foregroundbackground distinction unit 303 distinguishes between a background and aforeground through comparison between feature amounts of the backgroundmodel and feature amounts of the input image 210. In step S404, the maskimage generation unit 304 generates a foreground mask image 230 based ona foreground image that has been obtained as a result of the distinctionbetween the background and the foreground. Furthermore, the parametersetting unit 308 generates a background mask image 214 by cutting out apartial image 213 corresponding to a restricted area from the inputimage 210 and applying blur processing to the partial image. Note thatthe generation of the background mask image 214 may be executed any timebetween the obtainment of the image in step S401 and later-describedstep S406 (image composition), and may be executed in parallel withsteps S402 to S405.

Subsequently, in step S405, the background image generation unit 305generates a background image 220 from the background model. In stepS406, first, the background image generation unit 305 makes thebackground mask image 214 overlap the background image 220, and then theimage composition unit 306 places the foreground mask image 230 on thebackground image 220 through transparent overlap. Subsequently, in stepS407, the output unit 307 outputs the composition result. Next, in stepS408, ending of processing is determined; if processing is to be ended,software processing is ended through YES branching. Note that ending ofprocessing is, for example, a user's instruction for stopping shooting.If processing is not to be ended, processing returns to step S401, andthe foregoing processing is repeated. It goes without saying that theforegoing processing is not dependent on a particular OS or platform.

FIG. 5 shows an example of a screen for setting a restricted areaprovided by the parameter setting unit 308 according to the firstembodiment. A setting dialog 500 includes an image display portion 510and a button arrangement portion 530. Any image, such as an initialbackground image and an input image, can be designated as an imagedisplayed on the image display portion 510 using another setting dialog(not shown). Note that an initial background image is a background imagethat is generated first by the image obtaining unit 301 since the startof obtainment of a series of images, and may be an image that isobtained first by the image obtaining unit 301.

The user can operate an area designation control 521 by pressing an addbutton 531 or 532. The area designation control 521 is composed of astart point 522 and a plurality of vertices 523, and the user candesignate a polygonal area as a restricted area using them. The startpoint and vertices can be moved and added by designating on-screenpositions using a pointing device, such as a mouse. The polygonalrestricted area is confirmed by designating the start point 522 againusing a mouse pointer. The result thereof is displayed as a restrictedarea 520, for example.

In the present embodiment, there are two types of advantageous effectsof the restricted area. An advantageous effect of the restricted areathat is brought about by the add button 531 is the restriction of anupdate of a background image in the designated restricted area throughapplication of blur processing to the designated restricted area. Thismethod is the aforementioned method for generating a background maskimage 214, and generates the background mask image 214 by, for example,applying the blur processing to a partial image corresponding to therestricted area in an input image. On the other hand, an advantageouseffect of the restricted area that is brought about by the add button532 is the restriction of an update of a background image with the useof an image of the designated restricted area as-is. For example, if therestricted area 520 is designated on an initial background image, apartial image corresponding to the restricted area 520 in the initialbackground image is used as the background mask image 214.

Pressing a delete button 533 will delete the settings of a selectedrestricted area. A restricted area is selected by, for example,designating any position within the restricted area using a mousepointer. Pressing an OK button 541 will store the settings of therestricted area. Pressing a cancel button 542 will end the settingdialog 500 without storing the settings of the restricted area. In thiscase, the state of the settings prior to the activation of the settingdialog 500 is maintained.

FIG. 6 is a flowchart showing processing related to the setting dialog500. Once an instruction for activating the setting dialog 500 has beenissued, the parameter setting unit 308 performs initialization (notshown) and displays the setting dialog 500. Next, in step S610, theparameter setting unit 308 waits for input of an operation event. Oncethe operation event has been input (in the present example, once anoperation of pressing one of the buttons 531 to 533, 541, and 542 hasbeen performed), the parameter setting unit 308 distinguishes the event.

First, if it is determined that the event is a restricted area additionevent in step S611, the parameter setting unit 308 executes processingfor adding a restricted area in step S612. This is processing forgenerating a polygonal area through an operation of pressing the addbuttons 531, 532 and a subsequent operation on the area designationcontrol 521. Here, as processes related to mask types (blurring, noprocessing) are the same, they will be Illustrated collectively.

If the event is not the restricted area addition event, whether theevent is a restricted area deletion event is determined in step S613. Ifit is determined that the event is the restricted area deletion event,the parameter setting unit 308 executes processing for deleting aselected restricted area in step S614. This is processing correspondingto an operation of pressing the delete button 533.

If the event is not the restricted area deletion event, whether theevent is a restricted area selection event is determined in step S615.If it is determined that the event is the restricted area selectionevent, the parameter setting unit 308 edits (moves) the start point 522and vertices 523 of the area designation control 521 in response to auser's instruction in step S616. The restricted area selection eventoccurs as a result of designating an existing on-screen restricted area(e.g., the restricted area 520) using a mouse pointer. Note that therestricted area to be deleted in the aforementioned step S614 is, forexample, a restricted area that has been selected by the restricted areaselection of step S615.

If the event is not the restricted area selection event, whether theevent is an OK button pressing event (an event that has occurred as aresult of pressing the OK button 541) is determined in step S617. If theevent is the OK button pressing event, the parameter setting unit 308confirms the current settings on the setting dialog 500 in step S618.Then, the parameter setting unit 308 deletes the setting dialog 500 andends a dialog. If the event is not the OK button pressing event, whetherthe event is a cancel button pressing event (an event that has occurredas a result of pressing the cancel button 542) is determined in stepS621. If the event is the cancel button pressing event, the parametersetting unit 308 ends the dialog without updating the state of thesettings in step S619. Processing of step S610 and subsequent steps isrepeated if an operation event is input after the completion or duringthe execution of the processing for adding a restricted area (stepS612), the processing for deleting a restricted area (step S614), andthe processing for selecting a restricted area (step S616), or if it isdetermined that the event is the cancel button pressing event in stepS621.

According to the above-described first embodiment, in generating aprivacy protection image based on background subtraction, a restrictedarea, in which reflection of an update of a background image based on aninput image is restricted, is designated as an area that is alwaysprotected when displayed. In this way, for example, a person who ismotionless for a long time can be displayed with privacy protectionregardless of the result of distinction between a foreground and abackground.

<Second Embodiment>

Although the first embodiment has described a configuration in which theuser designates a restricted area on the setting dialog 500, a methodfor setting a restricted area is not limited to this. A secondembodiment describes, as another example of the method for setting arestricted area, a configuration that automatically adds a human area asa restricted area.

FIG. 7 shows an example of an initial state of the setting dialog 500according to the second embodiment. A difference from the firstembodiment lies in the provision of an automatic add button 734. If theautomatic add button 734 is pressed, human detection processing isexecuted with respect to an image displayed on the image display portion510, and detected human areas 711, 712 are displayed as candidate areas.In this state, if the OK button 541 is pressed, the parameter settingunit 308 registers the human area 711 and the human area 712 asrestricted areas. Furthermore, the user can delete an unnecessarycandidate area. For example, only the human area 711 can be set as arestricted area by designating the human area 712 and pressing thedelete button 533. There is no need for the user to specify a restrictedarea by operating the area designation control 521; this simplifies anoperation of setting a restricted area.

FIG. 8 is a block diagram showing an exemplary functional configurationof a video generation apparatus 300 according to the second embodiment.A difference from the first embodiment lies in that a human detectionunit 810 is added and a restricted area is set based on the result ofthe detection thereby.

FIG. 9 is a flowchart showing processing for automatically setting arestricted area, which is executed by the human detection unit 610 andthe parameter setting unit 308. Once the present processing has beenstarted by pressing the automatic add button 734, the human detectionunit 810 obtains an image in step S901. The image is an image displayedon the image display portion 510; here, it is an input image obtained bythe image obtaining unit 301. In step S902, the human detection unit 810detects humans from the image obtained in step S901. Then, in step S903,the human detection unit 810 executes tracking processing with respectto the detected humans. Here, human detection can incorporate a patternmatching method using human images, a method based on feature amounts, amethod based on machine learning, and so on, and is not dependent on aparticular human detection method. Next, in order to prevent erroneousdetection, the human detection unit 810 extracts a human for whichtracking has been performed throughout a predetermined number of framesor more in step S904. In step S905, the parameter setting unit 308registers an area of the extracted human as a candidate area.

In registering a candidate area, the result of the tracking can bechecked, and the registration can be limited to a human with a movementamount smaller than a predetermined value. As a result, in FIG. 7 forexample, a periphery of the human area 711 corresponding to a motionlessperson can be automatically registered as a mask area, and the humanarea 712 corresponding to a moving person can be excluded from a target.

As described above, according to the second embodiment, in setting abackground image to be displayed in a privacy protection image, an areawith a high privacy protection effect can be easily set as a restrictedarea.

<Third Embodiment>

In the first and second embodiments, privacy is protected by using animage obtained by applying blur processing or mosaic processing to animage of a restricted area as a background mask image, or by using apartial image corresponding to a restricted area in an initialbackground image as a background mask image. In the third embodiment,privacy is protected in a restricted area by setting a longerbackgroundization time for a restricted area than for other areas.

That is to say, in the third embodiment, the privacy protection effectis made continuous by setting different backgroundization times fordifferent areas. For example, an area in which a motionless person doesnot currently exist within an image but could possibly exist in thefuture, such as an area around a chair, is set as a mask area, and alonger backgroundization time is set for the mask area than forsurrounding areas. As a period from when a person sits on the chairuntil reflection of the person in a background image as a background islonger that in the surrounding areas, the privacy protection effectcontinues for a long time.

A functional configuration according to the third embodiment is similarto that according to the first embodiment (FIG. 3). Furthermore, asetting dialog is similar to that according to the first embodiment(FIG. 5). The following describes processing for an area that has beenset as a restricted area. As stated earlier, whereas the first andsecond embodiments protect privacy by processing a background image fordisplay (through overlap of a background mask image), the thirdembodiment protects privacy by changing a backgroundization time of abackground model on an are-by-area basis.

A backgroundization time will now be explained. According to abackground subtraction method, feature amounts are extracted from aninput image, and weighted and mixed with a background model. At thistime, a period until reflection in a background image can be extended byincreasing a contribution rate of an existing background. Therefore, abackgroundization time can be defined as a contribution rate of an inputimage with respect to a background model. Here, a completely fixedbackground is selected by making the contribution rate zero.

FIG. 10 is a flowchart showing background subtraction processingexecuted by the background model update unit 302 according to the thirdembodiment. In step S1001, the background model update unit 302 obtainsan input image. In processing of next step S1002, steps S1003 to S1006are executed with respect to all partial areas within the image. In stepS1003, the background model update unit 302 applies (selects) aprescribed backgroundization time as an initial value. Next, in stepS1004, the background model update unit 302 distinguishes whether atarget partial area belongs to a restricted area. For example, if 50% ormore of the partial area is included in the restricted area, it isdetermined that this partial area belongs to the restricted area. If thetarget partial area belongs to the restricted area, the background modelupdate unit 302 applies (selects) a backgroundization time for therestricted area (a backgroundization time longer than that for otherareas) in step S1005. In step S1006, a background. model of this partialarea is updated based on the set backgroundization time. In this way,for the partial area that belongs to the restricted area, abackgroundization time longer than that for other areas is applied.

As described above, according to the third embodiment, a longbackgroundization time can be set for a restricted area targeted forprivacy protection. This slows down backgroundization of a person who ismotionless for a long time in the restricted area, and as a result,privacy can be protected.

(Exemplary Modifications)

Although a longer backgroundization time is set for a restricted areaset by the user than for other areas in the third embodiment, it goeswithout saying that a restricted area may be set based on the result ofhuman detection as in the second embodiment. In this case, an area inwhich a human was detected over a period longer than a predeterminedthreshold is used as a restricted area, and a longer backgroundizationtime is set therefor than for other areas. Furthermore, although abackground mask candidate area based on human detection is obtained froma current video in the second embodiment, no limitation is intended inthis regard. A restricted area may be set by detecting a human from arecorded video of the past. In this case, in the recorded video of thepast, an area in which a cumulative value of periods in which a humanwas detected exceeds a predetermined threshold may be set as arestricted area. In addition, in the third embodiment, a longerbackgroundization time may be set for a restricted area that has beenset in the foregoing manner than for other areas. In this was,reflection of an image of a person in a background image is restrictedin an area in which a human was detected frequently in the past, andmore accurate privacy protection can be achieved.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, as optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2017-040922, filed Mar. 3, 2017, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image generation apparatus, comprising: aprocessor executing instructions stored in a memory that, when executedby the processor, cause the processor to function as: a generation unitthat generates a output image in which a foreground mask image overlapsa background image, the foreground mask image having been generatedbased on a moving object included in an input image, which is detectedby background subtraction method for the input image; an update unitthat executes backgroundization by update processing for updating atleast a partial area within the background image based on the inputimage so that the partial area which stays motionless over apredetermined backgroundization time is reflected in the backgroundimage; a setting unit configured to set a restricted area in whichreflection of the backgroundization is restricted; and an applying unitconfigured to apply at least on of blur processing and mosaic processingto the restricted area set in the background image, or to apply apartial image of an initial background image corresponding to therestricted area, as a partial image of the background image for theoutput image so as to mask an object in restricted area even if theobject stays motionless over the predetermined backgroundization time.2. The apparatus according to claim 1 wherein the setting unit sets therestricted area based on a user operation.
 3. The apparatus according toclaim 1 wherein the instructions further causes the computer to functionas a detection unit that detects a human area included in the inputimage, and wherein the setting unit sets a human area detected by thedetection unit as the restricted area.
 4. The apparatus according toclaim 1, wherein the input image is a streaming image being shot by animage capturing apparatus.
 5. The apparatus according to claim 1,wherein the input image is an image shot and stored by an imagecapturing apparatus in the past.
 6. An image generation apparatus,comprising: a processor executing instructions stored in a memory that,when executed by the processor, cause the processor to function as: ageneration unit that generates an output image in which a foregroundmask image overlaps a background image, the foreground mask image havingbeen generated based on a moving object included in an input image,which is detected by background subtraction method for the input image;an update unit that executes backgroundization by update processing forupdating at least a partial area within the background image based onthe input image so that the partial area which stays motionless over apredetermined backgroundization time is reflected in the backgroundimage; a setting unit configured to set a restricted area in whichreflection of the backgroundization is restricted; and an applying unitconfigured to apply, to the restricted area set in the background image,a first backgroundization time longer than a second backgroundizationtime applied to other areas in the background image, so as to keep anobject which stays motionless in the restricted area masked over aperiod longer than an object in the other areas.
 7. The apparatusaccording to claim 6, wherein the setting unit sets the restricted areabased on a user operation.
 8. The apparatus according to claim 6,wherein the instructions further causes the computer to function as adetection unit that detects a human area included in the input image,and wherein the setting unit sets a human area detected by the detectionunit as the restricted area.
 9. The apparatus according to claim 6,wherein the input image is a streaming image being shot by an imagecapturing apparatus.
 10. The apparatus according to claim 6, wherein theinput image is an image shot and stored by an image capturing apparatusin the past.
 11. A control method for an image generation apparatus, thecontrol method comprising: generating an output image in which aforeground mask image overlaps a background image, the foreground maskimage having been generated based on a moving object included in aninput image, which is detected by background subtraction method for theinput image; executing backgroundization by update processing forupdating at least a partial area within the background image based onthe input image so that the partial area which stays motionless over apredetermined backgroundization time is reflected in the backgroundimage; setting a restricted area in which reflection of thebackgroundization is restricted; and applying at least one of blurprocessing and mosaic processing to the restricted area set in thebackground image, or applying a partial image of an initial backgroundimage corresponding to the restricted area, as a partial image of thebackground image for the output image so as to mask an object in therestricted area even if the object stays motionless over thepredetermined backgroundization time.
 12. A non-transit computerreadable storage medium having stored therein a program for causing acomputer to execute a control method for an image generation apparatus,the control method comprising: generating an output image in which aforeground mask image overlaps a background image, the foreground maskimage having been generated based on a moving object included in aninput image, which is detected by background subtraction method for theinput image; executing backgroundization by update processing forupdating at least a partial area within the background image based onthe input image so that the partial area which stays motionless over apredetermined backgroundization time is reflected in the backgroundimage; setting a restricted area in which reflection of thebackgroundization is restricted; and applying at least one of blurprocessing and mosaic processing to the restricted area set in thebackground image, or applying a partial image of an initial backgroundimage corresponding to the restricted area, as a partial image of thebackground image for the output image so as to mask an object in therestricted area even if object stays motionless over the predeterminedbackgroundization time.
 13. A control method for an image generationapparatus, the control method comprising: generating an output image inwhich a foreground mask image overlaps a background image, theforeground mask image having been generated based on a moving objectincluded in an input image, which is detected by background subtractionmethod for the input image; executing backgroundization by updateprocessing for updating at least a partial area within the backgroundimage based on the input image so that the partial area which staysmotionless over a predetermined backgroundization time is reflected inthe background image; setting a restricted area which reflectionbackgroundization is restricted; and applying, to the restricted areaset in the background image, a first backgroundization time longer thana second backgroundization time applied to other areas in the backgroundimage, so as to keep an object which stays motionless in the restrictedarea masked over a period longer than an object in the other areas. 14.A non-transitory computer readable storage medium having stored thereina program for causing a computer to execute a control method for animage generation apparatus, the control method comprising: generating anoutput image in which a foreground mask image overlaps a backgroundimage, the foreground mask image having been generated based on a movingobject included in an input image, which is detected by backgroundsubtraction method for the input image; executing backgroundization byupdate processing for updating least a partial area within thebackground image based on the input image so that the partial area whichstays motionless over a predetermined backgroundization time isreflected in the backgroundization image; setting a restricted area inwhich reflection of the backgroundization is restricted; and applying,to the restricted area set in the background image, a firstbackgroundization time longer than a second backgroundization timeapplied to other areas in the background image, so as to keep an objectwhich stays motionless in the restricted area masked over a periodlonger than an object in the other areas.
 15. An image generationapparatus, comprising: a processor executing instructions stored in amemory that, when executed by the processor, cause the processor tofunction as: an obtaining unit configured to obtain an input imagecaptured by an imaging unit; a first generation unit configured togenerate a foreground mask image for obscuring an object detected fromthe input image; an update unit configured to execute backgroundizationby update processing for updating at least a partial area within abackground image based on the input image so that the partial area whichstays motionless over a predetermined backgroundization time isreflected in the background image; and a second generation unitconfigured to generate an output image by superimposing the foregroundmask image on the background image and by performing image processingfor obscuring a restricted area set in background image, so as toobscure an object in the restricted area even if the object staysmotionless over the predetermined backgroundization.
 16. The apparatusaccording to claim 15, wherein the second generation unit generates theoutput image by superimposing the foreground mask image on thebackground image and by performing at least one of blur processing andmosaic processing to the restricted area in the background image. 17.The apparatus according to claim 15, wherein the second generation unitgenerates the output image by superimposing a partial image of aninitial background image corresponding to the restricted area and theforeground mask image on the background image.
 18. The apparatusaccording to claim 15, further comprising a setting unit configured toset the restricted area in which reflection of the backgroundization isrestricted.
 19. The apparatus according to claim 18, wherein the settingunit sets the restricted area based on a user operation.
 20. A controlmethod for an image generation apparatus, the control method comprising:obtaining an input image captured by an imaging unit; generating aforeground mask image for obscuring an object detected from the inputimage; executing backgroundization by update processing for updating atleast a partial area within a background image based on the input imageso that the partial area which stays motionless over a predeterminedbackgroundization time is reflected in the background image; andgenerating an output image by superimposing the foreground mask image onthe background image and by performing image processing for obscuring arestricted area set in the background image, so as to obscure an objectin the restricted area even if the object stays motionless over thepredetermined backgroundization time.
 21. The control method accordingto claim 20, wherein the output image is generated by superimposing theforeground mask image on the background image and by performing at leastone of blue processing and mosaic processing to the restricted area setin the background image.
 22. The control method according to claim 20,wherein the output image is generated by superimposing a partial imageof an initial background image corresponding to the restricted area andthe foreground mask image on the background image.
 23. A non-transitorycomputer readable storage medium having stored therein a program forcausing a computer to execute a control method for an image generationapparatus, the control method comprising: obtaining an input imagecaptured by an imaging unit; generating a foreground mask image forobscuring an object detected from the input image; executingbackgroundization by update processing for updating at least a partialarea within a background image based on the input image so that thepartial area which stays motionless over a predeterminedbackgroundization time is reflected in the background image; andgenerating an output image by superimposing the foreground mask image onthe background image and by performing image processing for obscuring arestricted area set in the background image, so as to obscure an objectin the restricted area even if the object stays motionless over thepredetermined backgroundization time.